Implicitly-Parallel Functional Dataflow for Productive Cloud Programming on Chameleon

One solution that makes parallel programming implicit rather than explicit is the dataflow model. Conceived ~35 years ago, it has only recently been made practical through systems such as Dryad and Swift [1]. We believe that we have successfully created a base for an implicitlyparallel functional dataflow programming model, as exemplified by Swift, a workflow language for executing scientific applications. This model has been characterized as a perfect fit for the many-task computing (MTC) paradigm. Some broad application classes that fit the MTC paradigm are workflows, MapReduce, high-throughput computing, and a subset of high-performance computing. MTC emphasizes using many computing resources over short periods of time to accomplish many smaller computational tasks (both dependent and independent), where the primary metrics are measured in seconds. MTC has proven successful in grid computing and supercomputing, but the distributed nature of today’s cloud resources pose many challenges in the efficient support of MTC workloads. This work aims to address the programmability gap between MTC and cloud computing, through an innovative parallel scripting language, Swift, which will enable MTC workloads to efficiently leverage cloud resources. This work will enable a broader class of MTC applications to leverage cloud systems.